Skip to main content

View Post [edit]

Poster: Evanderro Date: Nov 2, 2016 5:55am
Forum: faqs Subject: Page cannot be displayed due to robots.txt

This topic is about some websites that are dead for a very long time and I noticed some strange things Web Archive is doing to them.

I used to check a few dead websites (their servers were permanently closed) via Web Archive and eveyrthing was OK till today...

"Page cannot be displayed due to robots.txt."

Could someone explain me why I was able to see these websites for example few months ago / a year ago / two years ago?

Websites were correctly displayed here many years after their deaths, everyone could use Web Archive to see them. So why now they're all blocked?

Mentioned websites are:
blinkgaming.net
twilightseve.tk

Reply [edit]

Poster: Jeff Kaplan Date: Nov 2, 2016 6:56am
Forum: faqs Subject: Re: Page cannot be displayed due to robots.txt

http://blinkgaming.net/robots.txt shows Disallow: / so it would not replay in the Wayback.

Reply [edit]

Poster: Evanderro Date: Feb 14, 2017 11:06am
Forum: faqs Subject: Re: Page cannot be displayed due to robots.txt

Look at this website again:

http://web.archive.org/web/20130402173522/http://blinkgaming.net/

Now it displays correctly. That's what I said. Sometimes it is blocked by robots.txt and sometimes not. Why?